Repository navigation
[FlyDSL] fp8_mqa_logits: derive the split target from kernel occupancy - #5921
Conversation
🏷️ CI GuideRuns automatically on every PR:
Extended tests (opt-in via labels):
One backend per PR: PR title tags & labels: |
43150be to
0b84266
Compare
There was a problem hiding this comment.
The split change looks right. On gfx942 a fixed cu_oversub cannot fit every variant: mfma_r2_w4 is 8 blocks/CU at 16 heads and 2 at 128 heads, so the same multiple over-splits one and under-splits the other. cu_count * occupancy, then picking the s that minimizes ceil(grid_x * s / effective_cus) * (ceil(W / s) + overhead), is the right replacement. gfx950 staying on the old path is also right; the writeup shows its optima are not in the region this model covers.
I checked the fallback too. The measured table is the min over build flags, and a miss falls back to occupancy 2. Both under-estimate, so the first launch under-splits instead of launching an extra wave. After the first dispatch the AMDHSA metadata takes over, and that matches HIP on all 216 gfx942 and 308 gfx950 configs. Output staying bit-identical follows from the kernel: a grid.y split is a disjoint slice of the KV window, with no reduction.
vpietila-amd
left a comment
There was a problem hiding this comment.
In general looks good to me. I have some comments regarding the maintainability, but consider them as optional. However, please consider if the kernel occupancy calculation works correctly under HIP graph capture (compiled vs eager modes).
a764709 to
abec67b
Compare
ChuanLi1101
left a comment
There was a problem hiding this comment.
The new push only answers the review comments. The split formula is unchanged, gfx950 stays on the old path, and CI is green on MI300X and MI355X. Approving again.
5199a20 to
fbe14d0
Compare
|
Thanks @coderfeli for the help and discussion, addressed both comments:
Please don't merge yet: this needs FlyDSL 0.4.0, and aiter still pins 0.3.4.1, so gfx942 fp8_mqa_logits CI will be red until the pin is bumped. I'll update the PR once 0.4.0 is on PyPI. |
_auto_num_splits targets a fixed multiple of the CU count, but the number of
blocks a CU actually holds is a property of the compiled kernel, not of the
device. On gfx942 the mfma_r2_w4 variant is 8 blocks/CU at num_heads=16 and 2
at num_heads=128, so one cu_oversub constant is correct for one kernel instance
and wrong for the rest: at 16x CU it over-splits the low-occupancy kernels (19
splits where 7 is optimal) and under-splits the high-occupancy ones (3 where 8
is optimal).
Replace the constant with the effective CU count, cu_count * occupancy, and
pick the split that minimizes a wave-quantized cost model:
cost(s) = ceil(grid_x * s / effective_cus) * (ceil(W / s) + overhead)
Occupancy comes from the kernel itself. FlyDSL JITs, so there is no
hipFuncAttributes handle before the artifact exists, but the artifact carries
the AMDHSA metadata that determines occupancy; kernel_occupancy reads it and
applies the hardware allocation rules. That result equals
hipModuleOccupancyMaxActiveBlocksPerMultiprocessor on all 216 buildable gfx942
configurations and all 308 on gfx950. A measured table covers the first launch
of a shape, which necessarily asks before any artifact exists, and a
conservative default covers anything else; both fall on the under-estimate
side, which under-splits rather than oversubscribes.
The two arches allocate AGPRs differently and the model is parameterized for
it: CDNA3 shares one register file between VGPRs and AGPRs, CDNA4 splits them.
Summing them on gfx950 mispredicts the bkv256 variants (vgpr=452/agpr=200),
which the sum puts over the 512-VGPR file at occupancy 1 where the hardware
reports 2. Every gfx942 kernel in this op reports agpr=0, so the distinction
changes nothing there.
gfx950 keeps the fixed-oversubscription path. Its occupancy is verified, but
its own measurements do not justify switching: its auto-selected variants span
only two occupancy values against six on gfx942, its optima sit 8-24x past one
effective wave (load imbalance across causal windows, which this model does not
represent), and 32 of 35 shapes have four or more split choices within 3% of
optimal against a ~2% noise floor. Rationale recorded in _SPLIT_POLICIES.
Perf, gfx942/MI325X, over the shapes vLLM issues for chunked long-context
indexer prefill (seq_len pinned near 1024 by VLLM_SPARSE_INDEXER_MAX_LOGITS_MB,
seq_len_kv sweeping 16k-128k, causal), on top of ROCm#4963:
DeepSeek-V4.1-Flash (index_n_heads=64): 1.25x, and 1.14x over a flat 16x
GLM-5.3-Flash (index_n_heads=32): 1.19x, and 1.07x over a flat 16x
ROCm#4963 is roughly neutral on DSv4.1 across that family, so nearly all of its
1.24x combined gain comes from this change; on GLM the two compose to 1.74x.
On aiter's own --model-shapes grid (block-diagonal batch_causal) the same
change is 1.08x over main with no shape regressing beyond its measured spread.
block_overhead_tiles is fitted over 86 shapes spanning three regimes -- vLLM
long-context prefill, block-diagonal batch_causal, and wide-window causal --
scored with the regimes weighted equally rather than by sample count, since
counts reflect how much was measured rather than what matters. Fitted on wide
causal alone it lands at 3, which then costs up to 1.21x on the narrow windows
production actually issues, because per-block fixed cost is a far larger share
of the work at 32 tiles than at 1024. The joint optimum is 8 under equal,
vllm-weighted and production-only weightings alike, and the worst case falls
from 1.21x to 1.07x.
Output is bit-identical to both prior policies at every in-window position on
every shape; splitting a row's window across grid.y is pure parallelism.
aiter's own sweep passes 1590/1590 cases.
Adds op_tests/flydsl/gen_fp8_mqa_logits_occupancy.py, which regenerates the
table and re-checks the model against HIP, and 25 unit tests for the selection
logic, the occupancy fallback chain, and the per-arch register-file rules.
Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com>
Trim comments, cache the occupancy on the launcher and the split choice, read the public CompiledArtifact.ir, load HIP through load_hip_runtime, and add a test that reads the compiled kernel's metadata against the measured occupancy table. Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com>
Replace the measured occupancy table, the IR metadata parser and the HIP generator script with CompiledFunction.max_blocks_per_cu() (ROCm/FlyDSL#1259), compiled with meta tensors once per launcher. Falls back to the cu_oversub rule on a first call inside graph capture. Requires FlyDSL >= 0.4.0. Do not merge before aiter's flydsl pin is bumped to it. Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com>
Name the split search cap (_MAX_SPLITS), drop stale numbers from the _auto_num_splits docstring, and have the occupancy test call _max_blocks_per_cu directly instead of monkeypatching _auto_num_splits. Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com>
fbe14d0 to
0a8998f
Compare
|
Quick status check on this PR as of 22nd Oct on the FlyDSL dependency: flydsl==0.4.0 has been pinned on main since #6424 (merged 2026-10-14), so about 8 days and has remained pinned re-checking the work on MI325X with FlyDSL 0.4.0: --full 1590/1590, default 95/95, --model-shapes 54/54 ran today on CI: rebased against main and all checks pass on reviewer feedback: addressed all open items I think this is ready to merge if you're happy with how it looks now @coderfeli 🙏 |
Overview
On gfx942,
fp8_mqa_logitsnow picks how many pieces to split each row's KV range into (grid.y) from how many blocks of the compiled kernel actually fit on a CU, instead of a fixed guess. gfx950 is unchanged.Motivation
When there are too few query rows to fill the GPU, the kernel splits each row's KV range across extra blocks. Today the number of splits targets a fixed
4 x CU countblocks.How many blocks fit on a CU depends on the compiled kernel, mainly its register use:
mfma_r4_w4mfma_r4_w2mfma_r4_w4mfma_r4_w2So one fixed target over-splits some kernels and under-splits others.
Suggested Fix
max_blocks_per_cu()(added in [Enh] Add CompiledFunction.max_blocks_per_cu() FlyDSL#1259, in FlyDSL 0.4.0). It is read once per kernel and cached.block_overhead_tiles = 8is the per-block fixed cost, fitted on MI325X.No occupancy tables or direct HIP calls are added.
Test Plan
op_tests/test_flydsl_fp8_mqa_logits.py --scenario verifywith--full, the default lane and--model-shapes.op_tests/test_flydsl_fp8_mqa_logits_splits.py: the cost model and a check that FlyDSL reports occupancy before the kernel has launched.Test Results
MI325X (gfx942), FlyDSL 0.4.0, rebased on current main.
Correctness:
--full1590/1590, default 95/95,--model-shapes54/54, splits test 6/6. Output is bit-identical to main on every shape below.Kernel time (µs, lower is better):
Geomean: 1.15x at 32 heads, 1.12x at 64 heads.
One shape is slower: 4k x 32k at 64 heads (0.96x). There the cost model picks 1 split where 2 is better.